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ABSTRACT 


18  new  business  R&D  contracts  were  investigated  for  the  effects 
of  considerations  of  follow-on  potential  on  contract  performance  and 
for  the  impacts  of  these  factors  on  further  contract  acquisition. 
The  three  dimensions  of  contract  performance--schedule,  cost,  and 
technical--correlate  significantly  with  each  other,  and  problems  in 
each  area  correlate  with  the  growth  in  contract  costs.   These  con- 
tract problems  are  greatest  in  those  cases  where  greatest  potential 
R&D  and  production  follow-on  was  expected  at  the  time  of  original 
proposal  preparation.   This  suggests  that  companies  that  anticipated 
large  follow-ons  wrote  into  their  initial  R&D  proposals  promises  of 
unrealistic  time  schedules,  cost  estimates  and  technical  performance 
goals. 

The  general  likelihood  of  R&D  follow-ons  is  more  accurately  pre- 
dictable at  the  time  of  original  proposal  preparation  than  is  the 
likelihood  of  production  follow-ons.   However,  except  for  the  trivial 
cases  in  which  it  is  clear  that  no  follow-on  potential  exists,  defense/ 
space  marketing  men  appear  unable  to  predict  effectively  the  dollar 
magnitude  of  the  resulting  follow-on  contracts. 

The  greater  the  problems  encountered  during  contract  life,  the  less 
the  original  expectation  of  direct  follow-on  from  the  same  government 
agency  was  realized.   However,  this  was  counterbalanced  by  a  marked 
improvement  in  win  ratio  in  technically  related  competitions  conducted 
by  other  government  agencies. 


*  Associate  Professor,  Sloan  School  of  Management,  M.I.T. 
+  Research  Assistant,  Sloan  School  of  Management,  M.I.T. 


Introduction 

In  the  defense/aerospace  contracting  business,  research  and  devel- 
opment (R£cD)  and  production  "follow-on"  contracts  to  initial  R&D  con- 
tracts often  comprise  a  large  portion  of  the  typical  company's  total 
government  business.   Consequently  consideration  of  possible  "follow-ons" 
plays  a  vital  role  in  a  company's  R6eD  marketing  decisions.   In  particular, 
many  R&D  contracts  are  of  an  investigatory  or  feasibility  study  nature, 
with  the  expected  follow-on  being  the  real  reason  for  the  initial  bidding. 
In  addition,  to  stay  in  business  a  defense/aerospace  firm  must  keep  up 
with  new  technologies  and  establish  market  positions  in  those  technical 
areas  likely  to  grow  significantly.   Yet  the  R&D  contracting  business 
is  characterized  by  the  high  cost  of  breaking  into  a  new  technical  area. 
To  some  extent  this  is  caused  by  widespread  competitive  attempts  to  win 
new  business  contracts,  even  at  high  costs  to  the  competing  companies, 
when  large  follow-on  is  expected.   The  rewards  for  the  high  entry  costs 
are  often  the  expected  follow-on.   Knowledge  and  hardware  gained  through 
performing  government  sponsored  research  are  salable  commodities  to  other 
government  agencies  and,  in  the  form  of  subcontracts,  to  industry.   Thus 
the  role  of  follow-on  contracts  should  be  of  primary  concern  to  those 
seeking  to  understand  the  performance  of  defense/space  companies. 

Summary  of  Original  New  Business  Study 

In  1965  a  study  was  performed  by  staff  members  of  the  M.I.T. 
Research  Program  on  the  Management  of  Science  and  Technology  to  inves- 
tigate successful  marketing  strategies  in  research  and  development  con= 


tracting.   During  the  research  effort  nine  cooperating  companies  sup- 
plied data  on  a  total  of  121  of  their  most  recent  competitive  situations 
that  met  the  following  sampling  criteria: 

1)  R&D  Research  and/or  development  was  the  primary  purpose  of  the 
contract. 

2)  New  Business   The  procurement  would  not  be  included  if  it  in- 
volved a  direct  follow-on  or  contract  extension  to  contracts  already 
held  by  the  company. 

3)  $100,000  or  Greater  in  Bid  Price  The  actual  bids  ranged  from 
$100,000  up  to  $10,500,000. 

4)  Prime  Contract  from  a  Federal  Government  Agency  Subcontracts 
were  excluded  from  the  sample. 

5)  Competitive  Procurement  A  procurement  was  included  if  the  RFP 
was  sent  to  at  least  two  companies,  regardless  of  the  number  submitting 
bids. 

6)  Contract  Awarded  Pending  and  cancelled  procurements  were 
excluded. 

It  was  the  intent  to  collect  information  on  about  twenty-five  contracts 
from  each  company.   Unfortunately,  far  fewer  suitable  contracts  were 


For  a  full  description  of  the  procurements  and  companies  involved,  see 
Kneissler,  Norman  W. ,  An  Investigation  of  Marketing  Strategies  in  the 
Government  R&D  Industry,  and  Ramsaier,  L.  Mark,  Performance  Measures 
and  Marketing  Strategies  in  Research  and  Development,  unpublished  S.M. 
theses,  M.I.T.,  February,  1966.   Copies  of  either  may  be  purchased  from 
the  M.IoTo  Microreproduction  Service  in  Microfilm  or  Xerox  form.   A 
brief  summary  of  the  above  studies  was  presented  by  Edward  B.  Roberts 
at  the  Boston  College  Conference  on  Marketing  in  the  Defense  Industry, 
May  19,  1966,  and  later  published  as  M.I.T.  Working  Paper  #235-66,  The 
Measurement  and  Improvement  of  R&D  Marketing  Effectiveness.  January,  1967, 


available  in  the  companies  studied.   Table  1  shows  the  actual  number  of 
competitive  situations  that  met  the  criteria. 

Table  1.   Disposition  of  the  Awards  by  Company 


Company 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Total 

Number  Won 

7 

2 

3 

1 

6 

1 

0 

0 

1 

21 

Number  Lost 

15 

10 

12 

9 

4 

17 

13 

17 

3 

100 

Total 

22 

12 

15 

10 

10 

18 

13 

17 

4 

121 

f 

In  the  original  study  only  one  question  was  asked  of  the  companies 

that  directly  treated  expectations  of  follow-on  contracts.   It  sought 

information  on  the  possible  impact  of  follow-on  considerations  on  the 

proposal  as  well  as  estimates  of  possible  R&D  and  production  follow-on. 

The  questionnaire  inquired: 

At  the  time  the  decision  was  made  to  bid,  did  you  particularly 
concern  yourself  with  potential  follow-on  which  might  result  from 
this  procurement?  Yes No 

If  yes,  what  was  the  total  dollar  volume  of  R  and  D  follow-on 
expected  from  this  procurement  in  the  five  years  following  re- 
ceipt of  the  RFP? 

(a)  From  this  customer:   $ ? 

(b)  From  other  government  agencies:   $ ? 


What  was  the  total  dollar  volume  of  production  follow-on  expected 
from  this  procurement  in  the  five  years  following  receipt  of  the 
RFP? 

(a)   From  this  customer:   $_ 


(b)   From  other  government  agencies  :   $_ 
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Ramsaier^  found  no  significant  relationship  between  follow-on  seekers 
and  contract  winners  (Table  2) . 

Table  2.   Effect  of  Expected  Follow-on  on  Initial  Awards 

Concerned  with 

Follow-on       Percent  Won    N    Significance  Level 

Yes  17.6       68 

.45 
No  17.5       51 

This  table  is  interpreted  as  indicating  that  17.67o  of  the  68  proposal 
teams  who  were  concerned  with  potential  follow-on  won  contracts.  A 
similar  percent  (17. 5Z)  of  the  51  "No"  answers  also  won  contracts. 

In  further  analyzing  the  original  data,  Dyer  has  found  a  number  of 
conclusions  concerning  follow-ons: 

(1)  The  mean  of  the  R&D  follow-on  expected  by  winners  is  statis- 
tically higher  than  the  mean  of  the  losers 

Mean  expected  R&D  follow-on,  winners  =  $23,440 
Mean  expected  R&D  follow-on,  losers  =  $  3,372 
(T-test,  1-tail  probability  =  .003)-^ 

This  fact  should  be  tempered  by  the  knowledge  that  the  data  were  collected 

after  the  contracts  were  awarded  and  the  winners  would  know  more  about 

the  agency's  plans  than  the  losers. 

(2)  Companies  place  a  higher  priority  on  contracts  with  expected 
follow-ons  in  that  they  are  more  likely  in  those  cases  to  establish  tech- 
nical contact  with  the  customer  prior  to  RFP  receipt  (p  =  .07). 


Ramsaier,  op.  cit . ,  p,  28. 

■^Rather  than  be  repetitive  from  this  point  on,  all  reported  probabili- 
ties will  be  for  one-tail  tests.   Thus  a  T-test  which  determined  a 
one-tail  probability  of  .003  will  be  shown  as  (T-test,  p  <»  .003). 


(3)   Those  proposal  teams  without  previous  experience  with  the  cus- 
tomer's technical  initiator  or  his  group  were  more  likely  to  bid  on  con- 
tracts if  there  was  expected  follow-on  (R&D,  T-test,  p  =  .08;  Production, 
T-test,  p  =  .14).   It  would  appear  that  companies  who  receive  an  RFP 
without  knowing  the  technical  initiator  feel  they  are  at  a  disadvantage. 
In  these  situations  they  mainly  go  after  those  contracts  that  will  give 
them  entry  to  new  technical  areas  that  have  good  potential  for  later 
government  R&D  and  production  follow-on  business. 

These  conclusions  initiated  our  interest  in  examining  follow-ons,  speci- 
fically with  the  twenty-one  winning  contract  situations  in  our  original 
study. 

The  Follow-on  Study 

In  August,  1966,  a  questionnaire  was  sent  to  each  of  the  21  re- 
spondents of  the  original  winning  proposals  with  the  purpose  of  learn- 
ing how  the  original  expectations  relate  to  the  actual  project  history 
and  follow-on  (so  far  and  now  expected),  and  how  the  bid-no-bid  decisions 
and  results  within  the  same  technical  area  have  proceeded  in  the  year 
and  one-half  since  the  awards.   Eighteen  questionnaires  were  returned 
fully  completed  with  two  incomplete  returns  and  one  non-response  (the 
latter  from  Company  9  which  only  bid  on  four  procurements  meeting  the 
criteria  during  the  original  test  period).   For  the  most  part  the  re- 
mainder of  this  paper  treats  the  analyses  of  the  eighteen  completed  ques- 
tionnaires from  six  different  companies. 
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The  questionnaire  contains  two  broad  divisions:   the  first  is  con- 
cerned with  the  performance  achieved  on  the  original  R&D  contract  and  the 
second  asks  questions  about  contract  follow-ons  received  up  to  the  date 
of  questionnaire  distribution  (August,  1966)  and  follow-ons  now  expected 
from  the  original  contract. 

Performance  on  the  Original  Contracts 

Fourteen  of  the  contracts  were  completed  as  of  questionnaire  response, 
with  the  other  four  estimated  at  more  than  90%  complete.   One-half  of 
the  eighteen  contract  situations  resulted  in  dollar  growth  ranging  from 
$1000  to  $694,000  while  the  remaining  nine  received  only  the  award  price 
(Table  3). 

The  growth  tended  to  be  found  in  the  larger  dollar  volume  contracts 
(contracts  for  over  $300,000).   The  growth  slightly  correlated  with  the 
difference  between  the  original  bid  and  the  negotiated  award  price,  al- 
though surprisingly  this  relationship  was  not  significant  (p  =  .34). 
Growth  was  not  a  function  of  contract  type  as  the  average  growth  per 
contract  was  about  equal  for  the  eight  fixed  price  and  10  cost-plus  con- 
tracts in  the  sample.   Many  of  the  non-responses  in  the  profit  columns 
are  attributable  to  fixed  price  contracts.   This  problem  and  the  fact 
that  four  contracts  were  not  complete  suggested  that  no  statistical  tests 
be  performed  on  the  profit  data.   But  these  data  are  presented  in  Table  3 
for  the  reader  to  draw  his  own  conclusions. 

The  managers  of  the  projects  had  had  prior  orientation  to  their 
jobs,  serving  in  all  but  one  case  as  head  of  the  proposal  team.   Even 
the  manager  of  the  single  exception  had  been  a  contributor  to  the  pro- 
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posal  effort.   This  background  of  the  manager  agrees  with  the  results 
found  by  Rubin  and  Marquis  in  a  study  of  forty-eight  government  sponsored 
R&D  projects.   Despite  the  project  managers'  experiences,  many  changes 
occurred  during  the  projects. 

In  one-third  of  the  contracts  technical  specifications  were  signi- 
ficantly altered  during  the  life  of  the  contract,  two  requiring  no  time 
extension.   In  one  case  significant  changes  in  the  program  direction  were 
informally  agreed  upon  to  resolve  differences  of  opinion  between  the  two- 
government  technical  monitors.   In  other  cases  increased  work  and  new  know- 
ledge gained  external  to  the  contract  were  cited  as  change  agents. 

Only  two  contract  situations  indicated  technical  problems  as  the 
reason  for  time  extensions,  while  the  rest  gave  the  customer's  expansion 
of  contract  objectives  as  the  cause. 

In  order  to  help  us  assess  the  nature  of  difficulties  encountered 
during  the  project  the  respondents  were  asked  to  rate  on  a  five-point 
scale  the  extent  of  problems  encountered  in  the  three  areas  of  schedule, 
cost  and  technical  performance.   The  resulting  data  are  displayed  in 
Table  4. 

The  three  contract  performance  measures  listed  in  Table  4  correlate 
highly  with  each  other.   A  non-parametric  (Kendall  Tau)  intercorrelation 
statistic  was  calculated  for  each  of  the  three  possible  pairs.   These  re- 
lationship  indices  are  shown  in  Table  5  with  the  corresponding  probability 
that  such  a  value  of  Tau  could  occur  randomly. 


Rubin,  Irwin  M-  and  Donald  G.  Marquis,  Critical  Decisions  in  the  Initia- 
tion of  Development  Projects,  M.I.T.  Working  Paper  #192-66,  June,  1966, 
p.  13. 


Table  4.   Contract  Performance  Ratings'*" 


Schedule 

Cost 

Technical 

Contract 

Problems 

Problems 

Problems 

A 

1 

1 

3 

B 

4 

2 

2 

C 

1 

4 

2 

D 

3 

3 

1 

E 

4 

4 

5 

F 

5 

5 

4 

G 

4 

3 

3 

H 

4 

4 

5 

I 

4 

5 

4 

J 

1 

I 

2 

K 

2 

1 

1 

L 

3 

3 

1 

M 

1 

3 

1 

N 

2 

2 

2 

0 

2 

2 

1 

P 

2 

1 

1 

Q 

2 

3 

1 

R 

2 

1 

3 

+  The  higher  the  rating,  the  more  serious  the  problem 

(e.g.,  1  =  problems  not  significant;  5  =  problems  were  crucial) 


Table  5.   Relationships  among  Contract  Problems  and  Contract  Cost  Growth 

Tau,  partial 
correlations 


Tau 

Probability 

Schedule-Cost 

.510 

.009 

Schedule-Technical 

.364 

.044 

Cost-Technical 

.300 

.088 

Schedule-Growth 

.228 

.165 

Cost-Growth 

.496 

.010 

Technica 1-Growth 

.186 

.195 

-.047 
.445 
.055 


The  three  measures  were  also  correlated  with  the  total  growth  in  the 
dollars  received  from  the  contract  (from  Table  3).   As  expected,  contract 
growth  correlated  most  highly  with  cost  problems.   In  fact,  because  of 
the  interdependences  among  the  various  performance  measures,  they  were 
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each  correlated  with  contract  growth  with  the  other  measures  partialed 
out  (i.e.,  the  effects  of  variation  by  the  third  and  fourth  variables 
upon  the  relation  between  the  two  variables  are  eliminated) .   The  Tau 
values  for  the  partial  correlations,  as  listed  in  Table  5,  indicate  that 
contract  growth  correlates  truly  only  with  cost  problems. 

When  asked  to  explain  the  nature  of  those  problems  rated  as  above 
average  in  seriousness,  respondents  indicated  that  cost  and  schedule 
problems  seemed  to  occur  most  frequently  when  the  technical  specifica- 
tions were  altered  without  adequate  changes  in  the  delivery  date  or  money 
awarded.   One  respondent  for  a  contract  that  achieved  low  technical  per- 
formance indicated  that  the  contract  was  essential  to  help  his  organiza- 
tion capture  a  much  larger  development  follow-on  contract  which  the  com- 
pany subsequently  lost. 

The  Predictability  of  Direct  Follow-on 

In  Table  6  the  direct  follow-on  dollars  received  and  expected  are 
compared  to  the  amounts  expected  approximately  at  the  time  the  original 
R&D  contract  was  awarded. 
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Table  6.   Direct  Follow-on  to  R&D  Contracts 

($000) 


R&D  Follow-on 


Production  Follow-on 


Originally 

Received 

Now 

Originally 

Received 

Now 

Expected 

So  far 

Expected 

Expected 

So  Far 

Expected 

0 

0 

0 

0 

0 

0 

5000 

1500 

3000 

0 

0 

0 

0 

0 

20000 

0 

0 

0 

0 

0 

0 

0 

0 

0 

200000 

30 

0 

0 

0 

0 

50000 

0 

0 

50000 

0 

0 

0 

0 

0 

500 

150 

150 

0 

0 

0 

350 

0 

150 

200000 

360 

23500 

50000 

0 

0 

0 

26116 

0 

0 

0 

0 

1500 

373 

1000 

0 

0 

0 

1000 

49 

100 

0 

30 

100 

800 

320 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

395 

500 

0 

0 

0 

0 

520 

500 

0 

0 

500 

500 

100 

500 

0 

0 

0 

If  the  magnitude  of  the  dollar  volume  is  neglected  and  follow-on  is 
momentarily  considered  as  a  something-or-nothing  variable,  the  following 
four  contingency  tables  can  be  set  up. 
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Table  7.   R&D  Follow-on,  Originally  Expected  vs.  Received* 


Received 
No  Follow-on 

Received 
Some  Follow-on 

Totals 

Expected 

No  Follow-on 

7 

3 

10 

Expected 

Some  Follow -on 

1 

7 

8 

Totals 

8 

10 

18 

*Fisher  Exact,  p  =  .02 


Table  8.   R&D  Follow-on,  Originally  Expected  vs.  Present  Expectations'*" 


Now  Expect 
No  Follow-on 

Now  Expect 
Some  Follow-on 

Totals 

Initially 

Expected 

No  Follow-on 

7 

3 

10 

Initially 

Expected 

Some  Follow-on 

3 

5 

8 

Totals 

10 

8 

18 

''Tisher  Exact,  p  =  .  l<i 
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Table  9.   Production  Follow-on,  Originally  Expected  vs.  Received"' 


Received 
No  Follow-on 

Received 
Some  Follow- on 

Totals 

Expected 

No  Follow-on 

13 

1 

14 

Expected 

Some  Follow-on 

3 

1 

4 

Totals 

16 

2 

18 

+  Fisher  Exact,  p  =  .37 


Table  10.   Production  Follow-on,  Originally  Expected 
vs.  Present  Expectations** 


Now  Expect 
No  Follow-on 

Now  Expect 
Some  Follow-on 

Totals 

Initially 

Expected 

No  Follow-on 

12 

2 

14 

Initially 
Expected  Some 
Follow-on 

2 

2 

.4 

Totals 

14 

4 

18 

**Fisher  Exact,  p  =  . 16 


The  high  statistical  significance  of  the  contrasting  data  arrayed 
in  Table  7  indicates  that  R&D  marketing  men  are  fairly  good  at  predict- 
ing whether  any  R&D  follow-on  (without  regard  to  amount)  will  result 
within  two  years.   Their  ability  at  longer  range  prediction,  however,  is 
not  as  good,  as  can  be  seen  in  Table  8.   The  results  suggest  that  the 
original  predictors  of  follow-on  are  still  generally  optimistic  about 
their  future  prospects  (neglecting  dollar  volume  considerations),  and 
they  are  reasonably  consistent  with  their  earlier  forecasts.   Tables  9 
and  10  and  their  corresponding  less  significant  probability  levels  in- 
dicate low  predictability  by  R&D  competitors  cf  possible  production  out- 
comes of  their  work.   Thus  we  can  say  with  a  reasonable  degree  of  con- 
fidence that  the  marketing  men  cannot  predict  production  follow-ons  at 
the  time  of  R&D  proposal  submission.   This  finding  suggests  that  possible 
production  follow-ons  should  not  be  weighted  heavily  when  fixing  a  bid 
price  in  an  R&D  competition. 

The  authors  feel  that  the  dollar  magnitude  estimates  provided  in  the 
questionnaire  responses  for  expected  follow-ons  merely  represent  orders 
of  magnitude  and  should  not  be  regarded  as  exact  interval  numbers  (i.e., 
parametric  tests  are  not  justified).   Therefore,  rank-order  correlations 
have  been  performed  on  the  original  expectations  against  the  follow-on 
money  received  so  far  and  that  now  expected.   (The  non-parametric  Kendall 
Tau  test  was  employed  in  these  analyses.) 

If  all  the  data  of  Table  6  are  used  (i.e.,  all  eighteen  contract 
situations),  the  original  R&D  follow-on  expectations  are  discovered  to 
be  strongly  correlated  with  that  received  so  far  (Tau  =  .34,  p  =  .07). 
In  addition  original  expectations  are  closely  matched  by  current  expecta- 
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tions  of  R&D  follow-ons  (Tau  =  .33,  p  =  .09).   However,  it  was  felt  that 
even  the  least  experienced  marketing  man  could  tell  from  the  nature  of 
certain  contracts  that  they  would  not  involve  follow-ons.   On  this  basis 
six  contract  situations  were  eliminated  in  which:   (1)  follow-on  expecta- 
tions were  $0  originally,  (2)  $0  has  been  received  so  far,  and  (3)  $0 
is  now  expected  to  be  received  in  the  future  (see  Table  6  for  these  data). 

The  remaining  contract  situations  are  concerned  in  some  way  with 
follow-ons.   Here  the  analysis  indicates  that  in  nontrivial  cases  the 
marketing  people  know  little  about  the  magnitude  of  expected  follow-ons. 
When  the  original  R&D  follow-on  expectations  are  correlated  with  the  follow- 
on  amount  received  so  far,  the  resultant  statistic  (Tau  =  "250,  p  =  .33) 
indicates  that  the  original  expectations  in  these  twelve  contract  situa- 
tions were  grossly  wrong,  even  as  to  the  order  of  magnitude  of  the  ex- 
pected research  and  development  follow-ons.   If  we  consider  the  sum  of  the 
present  expectation  of  future  R&D  follow-on  (remembering  that  the  original 
contracts  are  for  the  most  part  completed)  plus  the  money  received  so  far 
as  our  best  estimate  of  total  R&D  follow-ons,  we  find  that  the  original 
estimates  correlate  even  more  negatively  (Tau  =  -.272,  p  =  .118). 

It  is  of  note  that  R&D  marketing  people  do  not  seem  to  base  present 
expectations  on  money  received  so  far  (intercorrelation  coefficient  of 
Tau  =  .051).   Yet  each  correlates  highly  (and  to  the  same  extent)  with 
the  sum  of  the  two,  i.e.,  present  actual  plus  future  expected  follow-on 
(Tau  =  .504  and  Tau  =  .469,  respectively). 

The  correlation  of  original  expectation  of  production  follow-ons  with 
the  corresponding  actual  and  now  expected  is  even  more  negative.  Although 
only  six  contract  situations  were  affected  by  nontrivial  production  follow- 
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on  considerations,  the  company  that  originally  expected  a  $500,000,000 
production  contract  must  have  made  bid  decisions  different  from  what  it 
would  have  done  if  it  had  expected  $0.00,  which  it  does  nowl 

Follow-on  Expectations  vs.  Project  Performance 

It  is  likely  that  expectations  of  large  follow-on  contracts  may 
have  led  the  companies  into  making  far  too  optimistic  performance  pro- 
mises in  the  original  R&D  contract  proposals.   Such  promises  would  then 
augur  problems  in  the  resulting  projects.   We  shall  now  look  at  the  rela- 
tionships of  the  various  measures  of  performance  on  the  original  R&D  con- 
tract to  the  original  expectations  of  follow-on. 

In  the  questionnaire  the  companies  rated  on  a  five  point  scale  the 
extent  of  problems  encountered  in  the  original  R&D  contract  in  the  three 
areas  of  schedule,  cost,  and  technical  performance  (Table  4).   A  rating 
of  one  indicated  that  the  problems  were  not  significant  while  a  five 
meant  the  problems  were  crucial  to  the  project. 

We  first  looked  at  the  rank-order  correlation  of  the  three  perfor- 
mance measures  and  the  original  expectations  for  R&D  and  production  follow-ons 

Table  11.   Follow-on  Expectations  and  Performance  Problems 
Problem  Measure  vs.  Original  Follow-on  Expectations    Tau   Probability 

Schedule  Problems  vs.  R&D  Follow-on  Expected 

Cost  Problems  vs.  R&D  Follow-on  Expected 

Technical  Problems  vs.  R&D  Follow-on  Expected 

Schedule  Problems  vs.  Production  Follow-on  Expected 

Cost  Problems  vs.  Production  Follow-on  Expected 

Technical  Problems  vs.  Production  Follow-on  Expected   .531 


.422 

.032 

.276 

.108 

.255 

.147 

.589 

.029 

.554 

.034 

.531 

.037 

-17 

Examining  the  correlation  of  the  paired  variables  with  the  others  partialled 
out  reveals  that  all  the  above  relations  are  indeed  correlated  significantly. 
Based  on  these  statistical  evidences  we  can  conclude  that  those  companies 
that  expect  large  follow-ons  submit  initial  proposals  with  unrealistic 
time  schedules,  cost  estimates,  and  technical  performance  goals  in  order 
to  capture  the  initial  R&D  award.   These  overoptimistic  promises  then  in- 
duce serious  problems  during  the  actual  project  duration. 

Because  both  R6eD  follow-on  amounts  received  so  far  and  the  R&D  money 
now  expected  correlate  so  well  with  the  sum  of  these  two  follow-on  figures, 
It  was  decided  to  use  this  sum  (i.e.,  money  received  so  far  plus  dollars 
now  expected)  as  a  proxy  for  actual  eventual  total  follow-on.   The  com- 
panies were  divided  into  those  identifying  a  serious  performance  problem 
(rating  of  four  or  more)  and  those  with  less  serious  problems  (i.e.,  three 
or  less).   To  remove  the  scope  of  the  contract  as  a  tnajor  influencing 
parameter,  we  used  the  algebraic  difference  (i.e.,  +  or  -  )  of  the  origi- 
nal expected  follow-on  less  the  sum  calculated  above.   The  results  split 
on  the  seriousness  of  scheduling  problems  are  shown  in  Table  12. 


Table  12.   Schedule  Problems  and  Changes  in  Expected  R&D  Follow-On 
(Changes  shown  in  thousands  of  dollars) 

Problems  rated  4  or  more  Problems  rated  3  or  less 

+500  -20,000  +500 

+199,970  0  0 

+50,000  -26,116  -900 

0  +100  0 

0  +850  -1000 

+176,000  0  -100 

A  +500  indicates  that  one  company  which  rated  schedule  problems  as 
high  as  4  on  the  scale  originally  expected  $500,000  more  follow-on  than 
it  now  expects  (including  that  already  received). 
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Again  we  treat  the  numeric  follow-on  sums  as  indicative  only  of 
order  of  magnitude  and  therefore  use  a  nonparametric  test  to  avoid  the 
assumption  of  interval  data.   To  test  whether  the  two  independent  groups 
have  been  drawn  from  the  same  population,  we  apply  the  Mann-Whitney  U 
test.   For  schedule  problems  and  R&D  follow-on  the  calculated  U  is  11 
with  p  =  .009.   With  this  small  amount  left  to  chance,  the  researchers 
conclude  that  serious  schedule  problems  statistically  result  in  the  loss 
of  expected  R&D  dollars.   A  similar  analysis  of  the  schedule  problems' 
effect  on  loss  of  originally  expected  production  follow-on  resulted  in 
a  U  =  10,  p  =  ,007.   (The  direction  of  effect  is  obtained  from  inspec- 
tion of  the  data.)  Again  we  conclude  that  schedule  problems  deleteri- 
ously  affect  production  contract  follow-ons. 

In  like  manner  both  cost  problems  and  technical  performance  diffi- 
culties were  used  to  split  the  contract  situations  into  comparative 
groups.   For  both  measures  of  performance,  problems  result  in  lowered 
contract  follow=>ons.   The  detailed  results  will  be  summarized  in  Table  16. 

Contracts  in  the  Same  Technical  Area 

It  was  necessary  to  protect  against  double  counting  of  follow-ons 
or  contracts  won  in  the  given  technical  areas.   The  possibility  existed 
that  two  or  more  original  contract  awards  to  one  company  dealt  with 
similar  technology  and  that  the  follow-on  awards  attributed  to  both 
would  lead  to  double  counting.   In  order  to  insure  against  such  an 


Ssiegel,  Sidney,   Nonparametric  Statistics  for  the  Behavioral  Sciences 
New  York;  McGraw-Hill  Book  Company,  1956),  p.  116, 
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occurrence,  the  technical  areas  of  the  contracts  were  examined  as  well  as 
the  follow-ons  and  contracts  to  other  agencies.   Only  for  company  2  were 
the  technical  areas  the  same  for  two  contracts  and  the  company  expected 
and  received  nothing  in  both  cases. 

In  the  two  years  since  the  original  awards  were  made,  there  have 
been  fifty-two  procurements  conducted  in  the  remaining  sixteen  technical 
areas  represented. 

Table  13.   Contract  Opportunities  in  the  Same  Technical  Area 

Number  of  follow-on  procurements  conducted?         R&D  51 

Production  _1 

Total  52 

Number  of  sole-source  awards  to  you?     18 
Number  of  competitive  proposals?    22   Won?  22 
Number  unsolicited  proposals  submitted?   _9   Won?  _4 

In  some  cases  awards  are  still  pending.   Taking  all  companies  to- 
gether, the  technical  experience  gained  through  the  initial  contract  cer- 
tainly has  been  utilized  by  other  agencies  (or  industry).   Forty-four 
follow-on  or  related  contracts  were  awarded  to  the  six  companies  for  a 
total  value  exceeding  $28, 000, 000. 

The  companies  were  asked  to  breakdown  their  bidding  in  the  technical 
area  by  agency.  Agency  1  (in  Table  14)  represents  an  aggregation  of  the 
ones  each  company  considered  most  important  in  utilizing  the  technology. 
Agencies  2  and  3  are  aggregates  of  the  companies'  second  and  third  most 
cited  government  organizations.   Twelve  original  contract  situations  led 
to  bidding  activity  in  the  same  technical  area,  six  led  to  bids  submitted 
to  two  or  more  agencies,  and  two  led  to  bidding  to  more  than  three  agen- 
cies.  On  the  average  each  of  the  original  18  contracts  studied  has  led 
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to  company  bids  in  about  three  follow-on  procurements.   An  alternate  view 
is  that  the  12  original  probable  nominees  for  follow-on  have  each  produced 
about  four  follow-on  opportunities. 

Table  lA.   Breakdown  of  Bids  in  Same  Technical  Area  by  Government  Agency 
No.  of  Procurements   Agency  1   Agency  2   Agency  3   All  other   Total 


Bid 

27 

16 

6 

3 

52 

No  bid 

2 

1 

3 

0 

6 

Won 

22 

11 

5 

3 

41 

Lost 

4 

2 

0 

0 

6 

Not  yet  awarded 

1 

3 

1 

0 

5 

Procurement  Dollars 

(in  $000) 

Won 

27 

.000 

410 

156 

4 

28, 

,170 

Bid 

but 

lost 

3 

,360 

110 

0 

0 

3, 

,470 

Bid 

but 

no  award 

65 

? 

7 

0 

65+ 

The  fact  that  only  one  production  contract  has  been  let  within  two 
years  after  the  18  new  business  R&D  contract  awards  may  suggest  how  few 
R6eD  jobs  result  in  production  opportunities.   It  also  may  indicate  the 
time  delays  between  initial  funding  of  R&D  projects  and  the  use  of  the 
technology  gained  for  production  items,   (Perhaps  another  study  is  justi- 
fied in  a  few  more  years  to  examine  production  follow-on  obtained  by  then.) 
The  dollars  won  and  lost,  as  shown  in  the  bottom  half  of  Table  14,  may  be 
somewhat  misleading  in  that  one  contract  win  of  $26,000,000  and  one  loss 
of  $3,000,000  dominate  the  totals.   The  response  data  on  the  question 
about  dollars  won,  lost,  and  bid  but  not  awarded  were  poor  in  some  cases 
so  that  the  numeric  capture  ratio  (number  of  contracts  won/number  of  con- 
tracts awarded)  is  for  this  study  a  more  accurate  measure  of  competitive 
success  than  a  win/loss  dollar  ratio. 
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The  numeric  capture  ratio  of  41/47  =  87%  of  related  "follow-on"  com- 
petitions demonstrates  the  value  of  winning  an  initial  contract  to  gain 
work  experience  and  possibly  a  reputation  in  a  given  technical  area.   Even 
assuming  eventual  loss  of  the  five  awards  still  not  announced  produces  a 
win  ratio  of  41/52  =  79%.   The  numeric  capture  ratio  of  the  original  study 
was  21/121  =  17%.   Even  eliminating  from  the  original  study  data  the  three 
companies  that  were  not  included  in  this  follow-on  analysis  gives  a  ratio 
in  the  initial  study  of  only  20/87  =  23%.   The  six  companies  in  the  pre- 
sent sample  have  the  following  capture  ratios: 

Table  15.   Numeric  Capture  Ratios* 
(number  of  contracts  won/number  of  contracts  awarded) 

_1  2_  3  4  5  6   Mean 

Original  c£|pture  ratio  (%)     32  16  20  10  60  5   23 

Follow-on  capture  ratio  (%)    89  0  80  78  91  100   87 

*Mann-Whitney  U  =  16,  p  =  .05 

Thus  only  company  2  has  not  yet  gained  in  overall  competitive  effective- 
ness from  the  government-sponsored  research  performed  during  its  original 
contract.   In  that  case  the  reason  is  that  no  new  contract  yet  has  been 
awarded  in  the  technical  areas  involved.   These  data  are  biased  somewhat 
by  including  sole-source  awards  in  the  follow-on  comparisons,  even  though 
they  were  excluded  in  the  initial  study.   However,  even  in  the  competitive 
follow-ons  the  win  record  was  22  out  of  30  awards  for  an  overall  capture 
ratio  of  73%,. 
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Summarizing  the  Effect  of  Performance  Problems  on  Follow-on 

We  are  now  ready  to  examine  the  effect  of  poor  performance  on  the 
loss  of  R&D  and  production  follow-on  to  the  original  government  contract- 
ing agency  and  to  all  other  government  agencies  (and  industry).   The  Mann- 
Whitney  U  statistical  test  was  used  with  the  following  results: 

Table  16.   Impact  of  Project  Performance  on  Follow-ons 

Schedule  Problems 

1.  Serious  problems  cause  loss  of  expected  direct  R&D  follow-on 
(U  =  11,  p  =  .009). 

2.  Serious  problems  cause  loss  of  expected  direct  production 
follow-on   (U  =  10,  p  =  .007). 

3.  Serious  problems  cause  no  statistically  significant  effect  on 
win-loss  ratio  or  number  of  contracts  won  in  similar  technical 
areas  for  different  government  agencies. 

Cost  Problems 

1.  Serious  problems  cause  loss  of  expected  direct  R&D  follow-on 
(U  =  18,  p  =  .087). 

2.  Serious  problems  cause  loss  of  expected  direct  production 
follow-on  (U  =  12,  p  =  .023). 

3.  Serious  problems  cause  no  statistically  significant  effect  on 
win-loss  ratio  or  number  of  contracts  won  in  similar  technical 
areas  for  different  government  agencies. 

Technical  Performance  Problems 

1.  Serious  problems  cause  loss  of  expected  direct  R&D  follow-on 
(U  =■  13,  p  =  .063). 

2.  Serious  problems  cause  loss  of  expected  direct  production  follow- 
on  (U  =  8,  p  =■  .012). 

3.  Serious  problems  cause  no  statistically  significant  effect  on 
win- loss  ratio  or  number  of  contracts  won  in  similar  technical 
areas  for  different  government  agencies. 
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In  addition,  the  analysis  did  indicate  (though  not  statistically  signi- 
ficant at  the  .10  level)  that  the  lack  of  serious  problems  tended  to 
correlate  with  a  larger  win-loss  ratio  for  contracts  awarded  by  other 
government  agencies. 

Conclusions 

An  investigation  was  made  using  questionnaire  data  on  proposal  pre- 
paration decisions,  contract  performance,  and  follow-on  procurements  in 
18  cases  of  original  R&D  new  business  awards.   It  was  found  that  market- 
ing men  are  able  to  predict  the  general  likelihood  of  later  R&D  follow-on 
contracts  more  accurately  than  they  can  predict  production  follow-ons. 
However,  except  for  the  trivial  cases  in  which  it  is  clear  that  no  follow- 
on  potential  exists,  the  dollar  magnitude  of  the  resulting  follow-on 
contracts  cannot  be  predicted  effectively.   Despite  the  uncertainty  of 
these  predictions  consideration  of  the  potential  follow-on  appears  to 
strongly  affect  aspects  of  the  original  R&D  proposals,  and  the  resulting 
contracts. 

In  the  R&D  projects  studied  the  three  performance  measures  of 
schedule,  cost,  and  technical  performance  are  highly  intercorrelated. 
The  greatest  performance  problems  in  all  three  areas  arose  in  those  con- 
tract situations  in  which  the  original  expectations  for  direct  follow-ons 
were  highest.   The  companies  that  anticipated  large  follow-ons  thus  appear 
to  have  written  initial  R&D  proposals  with  unrealistic  time  schedules,  cost 
estimates,  and  technical  performance  goals.   The  problems  that  were  created 
by  not  meeting  these  three-dimensional  goals  have  caused  loss  of  expected 
direct  follow-on  contracts  in  both  R&D  and  production  areas. 
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Yet  it  seems  that  other  government  agencies  either  do  not  check  too 
closely  into  the  past  performance  of  the  company  or  are  not  adversely 
affected  by  performance  problems  in  contracts  performed  for  other  agencies. 
The  other  agencies  appear  to  be  interested  only  in  the  fact  that  the  com- 
pany has  experience  in  the  technical  area.   This  is  suggested  by  the  far 
higher  win  ratio  generally  achieved  by  the  companies  in  follow-on  areas 
than  in  the  areas  of  original  competition  studied  earlier.   Thus  in  de- 
ciding whether  or  not  to  "promise  the  moon"  in  a  new  business  proposal, 
R&D  contractors  need  to  balance  the  potential  loss  of  some  of  the  ex- 
pected direct  follow-on  from  the  government  agency  awarding  the  initial 
contract  against  the  increased  business  potential  that  becomes  available 
from  other  government  organizations  once  the  initial  contract  work  is 
performed.   This  situation  appears  to  be  unfortunate  to  the  interests  of 
both  government  and  effective  industrial  contractors.   It  would  be  desir- 
able that  good  R&D  performance  be  more  consistently  rewarded  and  poor  per- 
formance more  consistently  penalized  by  the  follow-on  process. 


Append ix--Questionna ire 


FORM  A   (WON)   FOLLOW-UP  ON  SOLICITED  COMPETITIVE  PROCUREMENTS 


Company_ 


Respondent^ 
RFP  No. 


RFP  Title 


Original  Contract  Form 


Title 


Company  Internal  Pro j .  No. 


Technical  Area 


Contracting  Agency_ 
Bid  Price  $______ 


Award  Price  $ 


Form  of  Final  Contract  (Please  check  if  renegotiated) 
^CPIF;  ^Fixed  Price; 


(Incentive)  ; 


CPFF; 


Fixed  Price 


Cost  Sharing; 


Other  (please  explain) 


I.   Project  History  (If  project  is  not  completed,  estimate  answers  where 
applicable.   Also  please  indicate  here  your  current  estimate  of  the 
percent  of  job  now  completed .^ ) 


What  is  the  total  amount  received  from  this  contract?   $_ 
How  much  growth  does  this  figure  include? 


In  Scope  $_ 


Beyond  Scope  $_ 


Original  Negotiated  Profit   $_ 
Final  Negotiated  Profit   $ 


2.   What  was  the  originally  contracted  completion  date?_ 


What  was  the  actual  completion  date? 

what  reasons  for  contract  extensions? 


If  different. 


1.  continuation  of  effort 

2.  expansion 

3.  technical  problems 

4.  other  (specify  please) 


3. 
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Were  the  technical  specifications  of  the  contract  altered  signifi- 
cantly during  the  life  of  the  contract?  Yes^ No 


How,  in  general? 


4.   Was  the  head  of  the  proposal  team  made  project  manager?  Yes No 

If  not,  who  was  made  project  manager? 


name  title 

Was  he  on  the  proposal  team?  Yes No 


5.   On  a  scale  from  1  to  5,  indicate  your  problems  in  the  following  areas- 
Schedule 

12         3         4         5 


problems 
not  significant 


problems 
were  crucial 


Cost 


problems 
not  significant 


problems 
were  crucial 


Technical  performance 


problems 
not  significant 


problems 
were  crucial 


6.   Explain  the  nature  of  tne  problem  in  the  areas  for  which  you  circled 
4  or  5  for  the  previous  question. 
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II-  In  order  to  learn  more  about  the  effect  of  winning  this  contract  on  your 
subsequent  bidding,  we  should  like  to  know  first  the  immediate  follow-ons 
coming  directly  from  this  contract  and  second  your  bidding  activity  within 
the  same  general  technical  area  as  covered  by  this  contract. 

1.   What  has  been  the  total  dollar  volume  of  direct  R&D  follow-on  from 
this  procurement  to  the  same  agency: 


so  far  $ 


expected  in  next  four  years  $ 


2.   What  has  been  the  total  dollar  volume  of  direct  production  follow-on 
from  this  procurement  to  the  same  agency t 


so  far  $ 


expected  in  next  four  years  $ 


3.   Taking  all  the  agencies  together,  please  answer  the  following  questions 
about  all  the  contract  opportunities  that  you  feel  to  be  follow-ons 
to  this  procurement. 

a.   How  many  follow-on  procurements  were  conducted?  R&D 

Production  .  " 


b.   Number  of  sole-source  awards  to  you 


c.   Number  competitive  solicited  proposals 
do   Number  unsolicited  proposals  submitted 


Number  Won_ 
Number  Won 


Break  down  your  bidding  in  the  area  by  agency  (e.g.,  NASA-LRC,  USAF-ESD, 
USA-Edgewood  Ars.) 


Noo  Procurements 

Bid 

No  Bid 

Won 

Lost 

$  Won 

$  Bid  but  Lost 

$  Bid  but  no  Award 


Agency  1. 


Agency  2. 


Agency  3 . 


All  Other 


Date  Due      -* 
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